The following relates to the language modeling arts, language processing arts, and related arts.
Language models are employed in natural language processing pipelines for applications such as speech recognition, machine translation, information retrieval, and so forth. Given an observed sequence of symbols, a language model predicts the probability of occurrence of the next symbol in the sequence. By way of illustration, in a language model for a written natural language, the symbols are typically words (e.g. in languages such as English or French) or characters or glyphs (e.g. in languages such as Chinese). In modeling of spoken language audio, the symbols are suitably audio segments corresponding to spoken words of the natural language.
Language models based on back-off smoothing have been developed which have demonstrated good predictive power, such as Kneser-Ney (KN) language models and variants thereof. Smoothing-based language models are heuristically based, and typically have numerous tuning parameters that are adjust in an ad-hoc fashion. Other approaches for learning language models have been proposed based on maximum entropy, conditional random field, or Bayesian approaches.